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ABSTRACT 



The primary purpose of this study was to provide evidence for ,or $ gainst the construct 
validity of the Career Skills Assessment Program (CSAP) instrument. A secondary purpds 
was to. present a systematic procedure for' carrying out internal construct validity 
studies in any testing instrument, tonstruct validation using confirmatory factor 
analysis indicated that the-CSAP instrument reliably measured what it purported to 
measure, and five of its six subscales provided sufficiently uriique information to 
make it a usefuVtool for program and/or individual diagnosis. 
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INTRODUCTION 

Construct validation attempts to establish evidence as to whether or not^ an instrument 
is measuring what it is purported to measure. Cronbach and Meehl (1955, p. 283) rfefine 
a construct as "some ^postulated attribute of people, assumed to be reflected in t^t 
performance." Well-known examples of constructs are verbal ability? quantitative 
ability, and spatial relations. The development of the Career Skills Assessment. 
Program (CSAP) as described in its handbook (College Entrance Examination Board, JjP* 
1978) Dostulated a six-construct performance model *o? career skills. The six per- 
formaffce areas are: (1) self-evaluation and development skills, (2) career-awareness 
skills, (3) career decision-making skills, (4) employment-seeking skills, (5) work- 
effectiveness skills, and (6) personal-economics skills. It was not necessarily assumed 
that all relevant career skills could be subsumed under a six-construct model but that 
the six areas represented a content core tha't was common to most developmental*' guidance J 
programs nationwide, regardless of the specific instructional materials being used. 

1 ** 

Since the Career Skills Assessment Program (CSAP) was designed to assess both in- , 

dividual and programmatic progress along the above six dimensions," any internal con- 
struct validation must demonstrate that (1) the individual constructs are internally K 
consistent and (2) eachymeasured area furnishes sufficient unique information to be 
able to make statements about individual and/or program strengths and weaknesses 
along ^these dimensions. These two construct validity criteria are critical^for for- 
mative evaluation. Tlje goal of formative evaluation is botfc individual and program 
improvement. It permits feedback to program staff and students on current progress 
along the measured dimensions, and often 'results in alteration or reemphasis of the * 
program or instructional sequence. 

The goals of demonstrating internal consistency as well as demonstrating differen- 
tiation among the measured career skills can best be represented as a problem in con- 
vergent and discriminant validity (Campbell and Fiske, 1959). Internal convergent 
validity suggests that items or subsets of items which were designed to measure a 
-particular career skills construct should "fit" or "load" on their appropriate factor. 
Given the "fitting" of such a pattern (i.e., a factor loading pattern 'consistent with 
the original test specif ications) * the question -then arises, what are the factor Shter- 
correlations giveft the hypothesized six skill area factor model? Discriminant validity 
provides some new information about strengths or weaknesses at either the individual or 
program level. 

* * * t* . 

It is the purpose of this research to marshall evidence «for or against the presence 
(or absence) of the six theoretical factors or constructs which are assumed to underlie 
the six item content areas. Alternative models which can be \hypothesized and tested 
range from the most parsimonious single-factor model to more complex multiple-factor 
solutions with varying degrees of complexity in between. Empirical evidence supporting 
a single-factor model would suggest that the reporting of six separate" scale scores 
would be* inappropriate. A fitted single-factor model would argue that the six scales 
-are congeneric measures (Lord and Novick, 1968) of a general career skij.1. That is, 
the true scores are perfectly cor relate^y From a* slightly different viewpoint, a 
single-factor model assumes* that the intelcorrelations among the otserved scale scores 
corrected for attenuation do not statistically differ from unity. Obviously, there 
would be no evidence for discriminant validity if t'he single-factor model held. 

Similarly, a factorlmodel pf more than one factor but less than six implies that at 
least one pair of observed scalp scores when corrected for attenuation do not statistic- 
ally differ from unity. Similarly to a single-fa.ctor outcome, the reporting of all six 
Separate scale scores wtfuld not be justified. yDther alternative models weight include a 
specification of more than six factors which include the^original hypothesized factors 
as a subset tod/or an entirely different factor structure than that assumed in the 
original instrument specifications. - \ ~" 
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To summarize, the alternative construct validity models are: (1) a single-factor 
^caree'r skill model, ("2) a reduced space career skill model (more than one factor but 
less than six), (3) a model of increased complexity which would include the original 
six constructs embedded in a larger factor model, and (i) an entirely different factor 
structure from that of the original test specifications. Tests of the above alternative < 
construct validity models can be carried out within the framework of confirmatory factor 
analysis. If confirmatory factor analysis results are not consistent with models 1 or 2 
above but are consistent with the original test specification^, then model 3 would be 
inconsistent with the empirical findings and model' 4 would be superfluous. 

*~\ That is, with respect to model 4, if the data confirm the original logically-derived 
specifications the're is no justification for trying to "fit" models outside of the 
original logical framework. However, if the data reject the original construct model, 
t then one must explore other models that may fit the data in hope that a post hoc lo^ic 
can be developed to explain such data-derived models. However, allowing the Sata to 
determine^ the model is not the classical approach to construct .validation and may be 
more aptly described as collecting and analyzing data in search of a construct. 



SAMPLE 

A total of 228 juniors and seniors from four different high schools ,had complete scores 
on the six career skills assessment scales. The high schools fere .located in the North- 
west and on the East coast. There were 137 juniors and 91 seniors. Although attempts 
were also made to gather reading scores on these same individuals, tlie subject who had 
all six scale scores and a reading measure was much too small to be used in this kind of 
analysis. -* # , 



METHOD 

Sorbom and Joreskog's (1976) program for confirmatory factor analysis was used to test 
(l)*a single-factor model afid (2) the a priori construct model defined by the original 
logical framework underlying the six CSAP^scales. 'The six-factor construct validation 
was carried out under two different specifications. The first specification will be 
referred to as a "true" or parallel score factor specification and the second will be 
called a congeneric factor specification. ' Fitting the a priori model under two differ- 
ent specif icat ions 'leads to a stronger case for or against the internal convergent and 
discriminant validity of the CSAP. The "true" score factor specification should approx- 
imate an "upper bound" on the internal consistency reliability of the fitted factors 
while the congeneric factor solution shoul^y^eld a "lower bound" estimate of the 
factorially defined scales' internal cojiSAStency . If the internal* consistency part 
of the model f f ts^ £g|ajffi ^;pnr Validity) then the level of intercorrelat ions among the 
factors- indicat^^ _ information inherent in each factor and thus estimates in 
a sense the disc^^S BP^ <iit:y of t ^ e faCtors - , l f tne a priori six-factor model is 
not in consist en r^^^M^^S^a then the relationships between year in school (junior or 
senior) and each ^^^^^^^»rg may be assessed. This type of analysis will indicate 
which, if ally, of .m^p^" fn1 1 " f T * a developmental pattern. That is, certain scales 
may be related to S^S^^prat ion % and for school curriculum. Appendix A gives a, technical 
discussion of the maximum- livelihood confirmatory factor analysis model used in the 
various tests of thie hypothesized factor patterns. 
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FIGURE 1. Single-Facfor Model Solution 

* Self-evaluation and Development; S^ ■ Career awareness; 
S^ = Career decision making; - Employment seeking; 

■ Work effectiveness; - Personal* economics. * 

.X 2 with 54 df » 714.68; P * 0.00; X 2 /df = 13.23; 
root mean square residual ■ .116. 

?1 



RESULTS AND DISCUSSION 



Single- Factor Model 



Figure 1 presents the factor ^loadings and goodness-of-f it tests for a model which 
assumes a single factor underlying the CSAP scales. The reader will note that there 
are 12 rather than six facCof loadings. 'S 
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The 12 vWables. result from splitting each of the six scales into odd and even 
^iVem halves. The splitting of each scale into tw6 halves allows for the possible pres- 
ence of odd-evfin correlations within scales whicK are not explained by the single 
factor. Figure 1 sjiows that .while all odd-even halves representing their respective 
career-skill areas have statistically significant loadings* 1 on the general factor, 
career awareness and;eareer decision making share the most common variance with the 
-general factor. This would appear to be reasonable since'there could be a cause-and- 
effect relationship between these^wo skill areas. That is, to a certain extent, 
appropriate career decision making is conditional on a high level of career awareness. 

What is of interest here isr the non-zero correlations between Che residuals ' ' 
* (errors) for corresponding split halves. These n correlat,ions are shown in Figure '1 on 
the curved lines between the odd and even errors. For example, the correlation between 
residuals for the two indicWors oi self-evaluation is .34. One can interpret this 
correlation between any two odd-even residuals as ttie correlation between odd-even in- 
dicators of a particular career-skill area with the influence of the general factor 
partialled out. If indeed a single factor were the true factor model, these correlated 
residuals would be close to zero. The probability associated with the large chi-square 
suggests that the likelihood of observing the sample correlations given a single-factor 
model is quite low. However, experience has shown that maximum-likelihood ratio tests 
Qf goodness of fit tend to reject assumed models even when the residuals - are only 
trivially different from zero. The> ratio of the chi-square to its degrees of freedom- 
as well as the root mean square residual 2 are ^also reported in Figure 1 and ^ill be the 
primary index of goodness of fit for any given 'model. Inspection of Figure 1 indicates 
that the overall root mean square residual r:, = .116. However, the root mean square 

residual for just the correlated errors from corresponding odd-even halves within a 
CSAP skill area was .30, which leads to the .judgment of lack' of fit of the single- 
factor model. If the odd-even item clusters were different item types*, then these 
additional skill-area factors might be simply reflecting correlated method variance. 
Although this is nqt the case, there remains some danger that these correlated residuals 
are being generated at leasTrin part by temporary and/or artifactual sources of correla- 
tion often observed when correlating odd-even split-halved scores. v - *- 



Six-Factor A Pri ori Models ^ . v 

. : ^ 

As pointed out before, the test of the a priori six-fa^ct^r model will* be carried out 
wider two different specifications. The first and less rigorous test will use the 
appropriate odd-even pairs to identify each of the six career-skill areas (the so-cajlled 
true score model). As pointed out in the single-factor model discussion, correlations 
between odd and even halves are sometimes inflated W temporary** sources of covariance 
and thus could lead to defining factors with lit tie generality, across samples or 
measures. To minimize the possibility of over interpret ingysuch factors, a second 
congeneric six-factor solution was ^lso tested using skill-^rea subscales to identify 
factors. This helps to overcome temporary sources of covarliance inflation such as 
item responses that use the *same item stems, etc. This second six-factor model spe- * 
cification will be referred to as a 'congeneric factor model. 

* € • 

1. The maximum-®Lkelihood standard errors of the factor loadings ran from .09 to 
*rl0. These standard errors should be interpreted with caution since the correlation 
matrix was analyzed which, of course, arbitrarily constrains the sample, variances to 
be equal, to l.G. - 

2. Root mean square* residual r - Lfi Z r* /p(p + 1)1 * i < J. 
* - ii L ij J f 



TABLE 1. Six-Factor Confirmatory Model Based on Split-Halved fccale Scores' 
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► Table 1 presents the results based on the fitting of the "true" score six-factor 
model. Using the ratio hf x to degrees <j£ freedom criterion, this six-factor constrained 
solution fits approximately eight times as well, as the single-factor model. Using the 
criterion of root mean square residual, the improvement in fit over the single factor 
model is by a factor of 4.3. Since the average residual is not practically different 
from zero' and the. largest residual is ,07, it may be concluded that the assumption of 
a six-factor model based on the six skill areas i^s not inconsistent with the observed 

'.'•/ " • . ■ 

Although the a priori six factors seem to yield a good fit to the data, the question 
arises whether a lesser number of factors might also lead to art acceptable fit. That is, 
is it possible that one or more pairs of the factors defined by the scales are so highly 
correlated that they could be collapsed into a single Jactor without a significant in- 
crease in lack of fit? In^p&ction of the intercorrelation matrix aihong factor's in 
Table 1 indicates that the majority of the correlations are all moderately high but % 
only one, r^y is so high as to lead one to quest iSfri whether there is' indeed two separate 

factors. The maximum likelihood factor analytic solution yields a standard error for 
095 which leads to a p ■ ,05 confidence interval around which does not in* 



' r 23, of 



elude r 2 2 * 1.0. Thus, on a strictly statistical basis it would be inappropriate to 
collaps'e factors *2 and 3* 

*- * 

♦ t * 

The finding of a high correlation {corrected for attenuation) between career-aware- * 

ness skills and career decision-making skills is consistent with what was. found wheit tne . 
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residuals firom the single-factor solution were examined. What is encouraging here is 
. v that the factor intercorrelatic^fe (with the possible excepti^on of r^) are ^sufficiently 

low to suggest that the CSAP does meet the criteria for discriminant validity and thus 
can prdvide useful information about strengths and weaknesses a^ong at least five of the 
original postulated dimensions. "^"Similarly, the high fac^r loadings and "the near-zero " 
residuals may be consider edv^idence for the convergent validity t6r CSAP. % * * 

N Stronger evidence for or against the discriminant and convergent validity of -the 

CSAP scales was -gathered by tatting the a_ priori six-factor , model using more than tyo 
measures to identify each factor apd to select these multiple measures so that they 
minimize the presence of correlated^errors. This approach was carried out by using 
- the multiple subscales that def fnesiftrareas/ under each of the dimensions as indicators 
of their respective dimensions. The number of independent career-skill subareas within 
a each of the six dimensions varied from four subscales under career-awareness skills to 
seven subscales under career decision making, "work effectiveness, and personal-economics 
skills. 

* ' t " ' 

'The CSAP developers designed these subscales to be independent* measures of their 
j respective constructs. Unlike split-halved measure^ which are likely to approximate 

^ parallelism, these independent subscales are only assumed to be congeneric measures'of 

their respective,censtructs. Therefore,- the a^ceptan^e of the a priori six-factor model 
where the factors are defined by the assumed multipltfcongeneric /measures -*is not only a* 
more powerful test of the presence or absence of the •finally specified six > factors, j 
r but is also a test of * the congeneric nature of the ind^teident subscales. That is, 

this is a^more rigorous^ iest of the CSAP construct specification because (1) factors 
are well ovef identified since there are more than three indicators of each hypothe- 
; sized factors, and (2) the factor indicators are independent subscales whi<rti are un- 
likely to suffer from inflated intercorrelations because of the temporary and arti- 
factual sources of over^p that are likely to inflate correlations between split-halved 
measures. 

Table 2 presents the factor loading matrix and intercorrelations .among factors 
based on the six-factor model Jhere each factor is defined by four or more congeneric 
measures. In spite of the over identification of each factor by the .congeneric measures, 
the goodness-of-f it criteria based on both the ^ 2 to df ratio and the* root mean square" 
^ . residual suggest quite a reasonable fit. The congeneric model fits almost as well as 

J t;he "true score" or parallel forms mo^el when the root mean square residual is used as 

the criterion of 'fit, and when the X 2 CD df ratio is used *he congeneric moUel appears 
to fit even better than the true score model* Inspection of the loadings in Table 2, 
-indicate which, Subscales are the most valid or reliable Measures of their particular 
factor or construct. Almost all the subscales have quite high correlations (factor 
loadings) with the appropriate constructs. Only. one Indicator, the sixteenth subscale 
(the seventh on career decision making) has a correlation in the low Fifties (r =*.52) 
This subscale was composed of only four items (the smallest Scale among the seven) and 
has to do with taking the "appropriate starting actions." As in the true score model, 
thp factor intercorrelations shown in Table 2 are corrected for attenuation. The con- 
^gfcneric specification of the six-factor model replicates the same pattern ,of factor 
, intercorrelations found *in the true score model . lectors 2 and 3 have the largest 
correlation, buWaUke the other model specification the p * .05 confidence interval 
does contain a correlation of y l,0. 

' * i * 

As pointed out earlier, the selection oi appropriate career decision-making 
choices (scale 3) is conditional on careej^awareness or 'career knowledge (scale 2). 

% Thus» although the two t do^not appear to share the 'sa/rfe item content, their apparent 
cause-and-eff ect relationship does lead to high intercorrelations. In a case such 
as this, it may be reasonable to report separate scores since "causes and effects" 
are not logically the same thing, but any interpretation of the differences between 
these two seores »f or. diagnostic purposes would be questionable. More will be said 

^ about this when the factor rel iabilities ^are presented. 
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TABLE 2. Congeneric S£x-Factor Model 
i 
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0.827 
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0.0 
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• 














tion about sel-f 
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,0.0 
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if.O 
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.and experience to career choices 


0;0 
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J o.o 


0,0 


* 0.0 


7. 
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0.769 ' 


0.0 


0.0 


o'o 
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0.0 
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0.0 


O,0 


• % 9. 
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0.0 


0.585 


0.0 


0.0 


0.0 


0.0 


10. 


Defining the problem 


0.0 


0.0 


0.593 


0.0 


0.0 


0.0 


V n. 


Establishing an action plan 
'Clarifying values 


0.0 


0.0 


0.806 


0.0^ 


0.0 


0.0 




0.0 


0.0 


0.774 


0.0 


0.0 


0.0 
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0.0 


0.0 
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- 14. 
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0.0 


0.720 
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16* 
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21. 
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*> fco person's needs and interests 
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0.621 


•o.o 
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0.0 
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*o.o 
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Achieving^ effective working re- 
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0.0 


0.0 
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effectively 


0.0 


0.0 


0.0 


- 0.0 


0.701 


0.0 * 


27. 


Advancing on the job * 


0.Q 


0.0 


# o.p 


0.0 


0.637 


' 0.0 
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Planning job changes / 
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0.0 


o.o - 


0.0 


0.£33 
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Figuring your paycheck arid income 














tax ' kl 
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* 0.0 ' 


0.0 


0.0 


0.0 
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procedures 


0.0 


o.o . 


0.0 


0.0 


0.0 


6.664 


3J. 


Purchasing goods and ( services and 








o.o 








paying bljLls * ^ 


0.0 


0.0 


o.o 


0.0 


0.8i3 



TABLED* Congeneric Six-Factor Model (continued) 



32. 

33. 
34. 
35. 



ssWlc 



possesions 
Borrowing and using credit 
Understanding investment procedures 
Understanding bas^c economic ideas 



1 

0.0 
0.0 
0.0 
0.0 



0.0 
0. 
0. 
0.0 



0.0 
0.0 
'0.0 
0.0 



0.0 
0.0 
0.0 

o,o y 



0.0 
O.G 
0.0 
0.0 



0.700 
0;772 
0.735 
0.749 



Intercorrelations Among Factors 

1.000 • 
0.690 1.000 

0.72j^ 0.808 , 1.000 

0.610 .0.671 0.855 "i.000 

,0.458 0.526 0.655 0-.612 

.0.317 *0.454 0.566 0.499 



1.000 
0.551 



1.000 



xVdf = 1,318 J 

Root mean square residual = .056 



TABLE 3. Factor Model Reliabilities and Standard Errors of Measurement 



odels 



Self- 
Evaluation 



I 

Career 
Awareness 



Career 
Decision 
Making 





Rel. 


SEM 


Rel. 


SEM 


Rel. 

* 


SEM 


Parallel (true score) 


.90 


2.58 


.85 


2.94 


.89 


3.00 


Congeneric 


.$0 


2ft 58 


.84 


3.04 


.87 


3.26 


Grade 11* 


.91 


3.11 


.90 


3.21 


.91 


' 3.33 


Grade l£* 
No. or items 


.92 


3.12 


.88 


3.14 




3.21 


60 




60 




60 



















r 



Models 



Employment— 
Seeking Skilly' 



Work 
Effectiveness 



Personal 
Economics 





Rel. 


SEM 


Rel. 


SEM 


Rel. 


SEM 


Parallel (true Score) 


.88 


.3.12 


.86 


/i.lk 


.91 


2.84 


Congeneric 


.87 


3.^5 


* .87 


2.64 


.90 


3.00 


Grade .11* 


.91 


3.22 




2.83 


.87 


3.32 


Grade 12* 


, .90 


3.08 


.91 


2.83 


.90 


3.25 


Ho. of items 


70 




. '60 




60 





*KR-20 Reliabilities and standard errors of measurement as reported in the Career 
Skills Assessment Program handbook. » 
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Tabl^ 3 presents the reliabilities of the scales based on phe two hypothesized 
.factor models. Appendix B presents the equations for computing reliabilities ofv the 
factors when they are estimated from a Confirmatory .factor-analysis solution* As 
pointed out earlier, the congeneric model should yield a lower bound 'internal con- 
sistency estimate since it" minimizes the biasing influence of temporary and artifactual 
, sources of covar lances! -The reliabilities dn Table 3 are quite acceptable for all six 
iactorally-def ined scales. It is encouraging to note that there appears *to be little 
& shrinkage when going from the parallel measures (,true score) model to the congeneric 
model. The lowest reliability undei> both model specifications is associated with 
career awareness/ This finding is reasonably consistent with the KR-20 estimates 
which were reported in the Career Skills Assessment Program handbook (College Board, 
1978). In general,' scale reliabilities based on fitted truS^ffdore factor solutions* 
tend" to yield the same level of reliabilities as does KR-20 but tend to be slightly 
lower here since the population of students used in the present study is characterized 
by consistently smaller subscale variances. .Because of the differences in variances, 
the standard errors of measurement are the *more appropriate indicators for comparing * 
accuracy of measurement across populations and estimation techniques. The maximum- 
likelihood factor analytic estimation of reliability and standard error of measurement 
does, however, take into ^consideration information about cross-scale correlations in 
the estimation procedure. That is, the? factor loadings which det^rjnine the factor re- 
liabilities are "consistent" estimates s^nce they (phe loadings) depend on their inter- 
relationships with all other variabiles in the factor /Solution. 

- N - . - "' 

The previous finding of the relatively high intercorrelations between career aware- 
ness and career decision-making skills along with the somewhat lower factor reliability 
, for career awareness suggest that it might be advantageous to report a single s*core — 
career awareness and decision-making skills — that is-, to collapse the two scales and 
report* a five-dimensional career skills pattern rather than the present six- dimensional 
pattern. This would increase the* reliability* of the composite scale and also reduce 
some of the present redundancy. 

<* 

Table 4 presents the -factor extension correlations with grade in school. The cor- 
'relations are all positive but not significantly different from zero. In this sample, 
1 there appears to be little or no relationship between year in school and score level. 
This lack of relationship may be partly artifactual since there is some evidence (based 
on the subscale means) that some individuals may have been "topping-out" in this par- 
^ jticular sample; that is, the mean scores were within one standard deviation of a per- 
fect score on a number of subscales. This is, of course, consistent with the finding 
of smaller scale variances. 



TABLE' 4.. Factor Extension Correlations with Grade in School 



Career Work 
Self- Career Decision Employment Effec- Personal 

EVal. Awareness taking Seeking t'iveness Economics 



Grade in School * , ^ 

(10th or 11th) .08 .13 .05 .14 .17 14 



■ ( 



/>^The results of this study appear to contradict those of Grandy (1979) who concluded 
that a single reading factor could explain the correlations between self -evaluation, 
career awareness,* and career decision making. Some of the possible reasons for this 
% apparent paradox are: (1) that their model was not as completely identified since they 
had only three potential factors, and (2) that the 'reading factor become quite salient 
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because their sample was characterized by a low reading level. If the latter condition 
prevailed, score levels on, the scales would be conditionals, on having at least a minimum 
reading level. Judging from the mean scores^in the present sample, the reading levels 
were probably" more than sufficient. . 

Since Grand^ had information on only three ,subscales, two of which appear to 
collapse into one factor iri the present study, it is not surprising that they came 
to v the conclusion they did. The fact remains that the present study, using a dif- 
ferent population and a vastly over-identified model (congeneric model), suggests 
that there* are at, least* five factors and these five factors are consistent with the 
original test specifications. - * * 

SUMMARY AND CONCLUSION 

The purpose of this study was'to provide^ evidence ^f or or against the construct validity 
of the CSAP instrument. A second purpose was to* present a systematic procedure for 
carrying out internal construct validity studies oa any testing instrument. 

" , • y r 

The Career Skills Assessment Program was designed to assess both individualLand 
programmatic progress, along six dimensions. The *^ix a re a^ are": .(1) self -evaluation 
and developmental skills, (2) career-awareness skills, (3)* career decision-making 
skills, (4) employment-seeking skills, (5) work-effectiveness skills, and (6) personal- 
economics skills, A construct validation plan based on confirmatory factor analytic 
procedures was implemented that preseated evidence for (l)*the internal ■ consistency 
of factors which were fitted according to the original test specif feat ions and (2) the 
relative independence or uniqueness of five of the content areas. Two of the pre- 
specif ied content areas — career awareness and xareer decisi6n making — appeared to 
collapse into one, factor. Although there was some restriction in score variability 
in the present sample, the factor reliabilities were sufficiently high *o justify the 
use of five of the CSAP scales as a program diagnostic tool. - 

-3? 

In summary, the CSAP appears to reliably measure what it purports to measure, and 
five of six of its subscales provide sufficiently* unique information to make thej#5AP 
a useful tool for program and/or individual diagriosis. 

*jf * * 
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APPENDIX A: Confirmatory Factor Analysis 

Sorbom and Joreskog's (1976) program for confirmatory factor analysis across populations, 
COFAMM, was used to test various implicit assumptions about the construct validity of the 
Cai*eer Skills Assessment instrument. COFAMM assumes that a factor analysis model holds 
in each of the g populations under study. In this case, g = 1* sinae there is only one' 
population under study. 1 If x is defined as the vector of the p observed measures in 

group g, then x can be\ accounted for by k common, factors (f ) and p unique factors (z ).. 

^g - ^g 

The model in each population *is: 

x.*v+Af+z % t * (1) 

where v is a p x 1 vector of location parameters and A a p x k matrix of factor loadings. 

yz ^g 
It is assumed that z and f are uncorrelatedV the expectation of z - 0 and the expecta- 

tion of f * 8 , where 8 is a x 1 parameter vector. 

Given these assumptions, the mean vector u of the x is • \ * • 

u = v + A 8 (2) 
^g ^g ^8^8 

and the expected variance-covariance matrix Z of x is 

^g ^g 

\* 

£ - A * A 1 + f ' (3) 
^g* ^g^g^g ^ / 



where $ is the variance-covariance matrix of the f and V is the variance-covariance • 

^g „ ^g *■ ^g 

matrix of z . When the factor model does not fit the data perfectly, thfc observed 

variance-covariance matrices S and observed means will differ from the maximum likelihood 

^g 

estimates of X and u The program yields a chi-square statistic that is a measure of 
^g 

these differences, that is,* of how well the model fits the data. 

The four matrices 6 , A , $ , and Y are called the pattern matrices* The elements 
^g ^g ^g ^g ~~ 

of these matrices are the model parameters, which are of three kinds: (a) fixed 

parameters, which have been assigned given values, like 0 or 1, (b) constrained 
parameters, which are unknown but equal to one or more other -parameters,, and (c) free 
parameters, which are unknown and not constrained to be equal to any other parameter. 
\ • A parameter may be constrained to be equal to other parameters in the same and/or dif- 

ferent' pattern matrices in the same and/or in different groups. 

^ An important featiire of a confirmatory analysis is that the parameters of the model 
may be uniquely estimated, i.e., the model is identified. A solution is unique if all 
linear transformations of ^;he factors that leave the fixed parameters unchanged also 
leave the free parameters unchanged. It is difficult in general to give useful condi- 
tions which are sufffcient for identification. However, at one point in the program the 
information matrix for the unknown parameters is computed. If this matrix is positive 
definite, it Is almost certain that the model is identified. If this matrix is not 
positive definite, t\\e program prints a message to this effect, specifying which 
parameter is probably not identified. * 
- * t * 

In all tests of the posited CSAP factors the models »we re over identified, yielding 
not only unique solutions but sufficient degrees of f reedofa for statistical tests of 
goodness of fit. If the model is identified, or over identified as in these examples, 
standard errors for all the unknown parameter estimates are ai^so provided by the 'program. 
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APPENDIX A: Confirmatory Factor Analysis (continued) 



"I In exploratory factor analysis, the model identification often depends on arbitrary 
restrictions which may have little to do with how the data were gathered. In the con- 
firmatory approach used here, we progressively constrained parameters in equation 3 be- 
ginning with the true score model and then the/xomgeneric model. The important point 
here is that these constraints are applied within the framework of the original test 
specifications and the nonrejection of each successively stronger model marshals more 
evidence for the construct validity of the CSAP. Conversely, rejection of the true 
score modej. and/or the congeneric model casts doubt on whether the test is measuring 
the constructs underlying tfie test specifications. 
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API^ENDIX B: Confirmatory Factor Analysis Estimates of Reliability 
I. Reliability of a CSAI subscale 

, . AA • ■ 

ii ,2 T 2 (!) 
ij 9 j ii 

II. Reliability of a factorially defined CSAI scale 



(i 0 



.- -„ 2 * j 

r. = ' ■" - J 



* j + Z i 



(i "ij) 

actor loa 
2 

jth factor, and ¥^ - uniqueness of the _ith subscale* 



(2) 



Vhere A ij = factor loa din$ of the i.th subscalej on the jth factor, $7 * variance of the 

. .„2 • *» 
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APPENDIX C: Correlation Matrix Among the Six Split-Halved Measures 





1 


2 


3 


4 


5 


6 


1 


1.000 












' 2 


0.822 


1.000, 








< 


3 


0.582 


0.526 


1.000 








4 


0. 531 


0.499 


0. 740 


1 . 000 * 






5 


0.603 


0.570 


0.637' ' 


-0.595 


1.000 




6 


0.534 


0.517 


0.615 


0.613 


0.795 


1.000 


7 


0.407 


0.472 


0.486 


0.496 


0.481 


0.515 


8 


0.479 


0.550 


0.540 ' 


0.514 


0.530 


0.525 


9 


0.367 


0.370 


0.370 


0.400 


0.544 


0.498 


10 


0.331 


0.333 


0.385 


0.399 


0.492 


0.469 


11 


0.306 - 


0.281 


0.407 


0.382 


0.498 


0.512 


12 


0.211 


0.232 


0.341 v 


0.288 


0.423 


0.428 




7 


8 


9 


10 


11 


12 


" 7 


1 . 000 












*8 


0.789 


1.000 










9 


0.489 


0.423 


1.000 








10 


0.529 


0.479 


0.757 


1.000 






11 


0.384 


0.399 


0.467 


0.427 


1.000 




12 


0.432 


0.407 


0.448 


0.439 


0.-337 


1.000 



r 

< 
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APPENDIX D: Correlation Matrix Among Subscale Scores 
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1 


2 




3 


4 


5 


*6 


7 


8 


9 




10 


1 


1 .000 








• 
















2 ' 


0.607 


1 .000 






















3 


0.615 


0.648 


1. 


000 






« 












4 


0.573 


0. 627 


0. 


702 


1.000 
















5 • 


0.595 


0.667 


0. 


653 


0.621 


1.000 














6 


#0.455 


0.494 


0. 


469 


0.444 


0.*480 


i.opo 












7 


0.434 


0.412 


0. 


36V^ 




0.368 


0.606 


1.000 










8 


0.421 


0.482 


0. 


426 


0H&4 


0.505 


0.69£ 


0.630 


1.000 








9 


0. 356 


0. 300 


0. 


330 


0.360 


0.316 


0.464' 


0.459 * 


0.495 


1.000 






10 


0. 349 


0. 363 


— 0. 


386 


0.391 


0.342 


0.284 


0.361 


0.401 


0.324 


( 

■r. 


000 


11 


0.471 


0.534 


0. 


529 


0.486 


0.559 


0.507 


0.570 


0.577 


0.385 


0. 


455 


12 


0.355 


0.417 


0.451 


0.369 


0.477 


0.476 


0.555 


0.519 


0.358 


0. 


461 


13 


0.404 


0.437 


0. 


406 


0.442 


0,489 


0.453 


0.486 v 


0.499 


0.3&4 


0. 


432 


14 


*0. 287 


0. 349 


0. 


339 


0.346 


0.365 


0.368 


0:462 


0.462 


0.394 


0. 


515 


15 


0.313 


0.435 


0. 


373 ' 


0.300 


0.321 


0.438 


0.486 


0.424 


0.350 


0. 


394 


16 


0.260 


0.376 


0. 


380 


0.354 


0.396 


0.284 


0.339 ' 


0.361 


0.266 


0. 


295 


17 


0.237 


0.299 


0. 


287 


0.263 


0.324 


0.446 


0.466* 


0.439 


0.154 


0. 


263 


18 


0.352 


0.425 


0. 


414 


0.320 


0.448 


0.417 


0,502 


0*461 


0.290 


0. 


354 


19 - 


0.361 


0.485 


0. 


453 


0.411 


0.530 


0.444 


0.399 


0.429 


.0.249 


0. 


337 


20 


0.352 


0.416 


0. 


352 


0.257 


0.450 


0.469 


0.401 


-0.442 


0.256 


0. 


282 


21 


0.241 


0.359 


0. 


319 


0.192" 


0.327 


0.339 


0.299 


0.342 


0.1^0 


0. 


297 


22 


0.287 


0.311 


0. 


313 


0.271 


0.298 


0.367 


0.412 


0.373 


0.202 


0, 


229 


23 * 


0.262 


0.268 


0. 


2'83 


0.279 


0.312 


0.246 


0.341 


0.345 


0.166 


0. 


211 


24 


0.316 


0.318 


0. 


220 


0.271 


0.230 


0.237 


0.336 


0.304 


0.190 


0. 


231 


25 


0.247 


0.187 


0. 


208 


0.209 


0.211 


0.215 


0.332 


0.316 


0.126 


0. 


259 


9 A 
ZD 


n i Q/. 


U . ZUo 


0. 


174 


A IOC 

0. 1 85 


0. 196 


0. 241 


0.318 


0.302 


0. 138 


0. 


226 


27 


0.249 


-0.275 


0-317 


0.347 


0.317 


0.323 


0.360 


0.317 


0.254 


0. 


205 


28 


0.188 


0.224 


0. 


214 


0.173 


0.258 


0.148 


0.213 


0*230 


0.126 


0. 


211 


29 


0.099 


0.232 


0. 


174 


0.118 


0.247 


0.200 


0.323 


•0.305 


0.147 


0. 


242 


30 


0.121 


0.099 


0. 


090 


0.069 


0.140 


•0.135 


0.158 r - 


.0.201 


0.106 * 


0. 


149 


\31 


0.222 


0.248 


0. 


222 ,* 


0.221 


0.291 


0.269 


0.349 


0.420 


0.182 


0. 


327 


32 


0.234 


0.326 


' 0. 


253 


0.256 


0.305 


0.329 


0.365 


0.420 


0.233 


0. 


348 


33 


0.217 


0.213 


0. 


177 


0.139 


0.258 


0.252 


0.293 


0;351 


0.212 


0. 


224 


34 


0.027 


0.071 


0. 


046 


0.014 


,0.146 


0.076 


0.170 


0.196 


0.050 


0, 


137 • 


35 


0.226 


0.258 


0. 


206 


0.214 


0.285 


0.263 


0.288 


0.343 


0.172 


0. 


320 



n 

12 
13 
14 
15 
16 
17 
18 
19 
20 
21 
22 
23 
24 
25 



11 

1.000 

0.652 

0.576 

0.534 

0.509 

0.454 

0.364 

0.440 

0.41 

0.38j 

0.331 

0.456 

0.411 

0.403 

0,330 



12 



1.000 
0.512 
0.582 
0.522 
0.429 
0.387 
0.5'04 
0.388 
0.413 
0.327 
0.378 
0.369 
0.291 
0.306 



13 



14 



15 



16 



17 



1'8 



19 



21 



20 



1.000 


















0.541 


1.000 


1.000 














0.553 


0.532 














0.297 


0.421 


J). 333 


1.000 












0.305 


0.318 


0.347 


0.15£ 


1 


000 








0.417 , 


0.431 


0.377 


0.243 


0 


613 


1.000 






0.582 ; 

0.357 


0.363 


0.353 


0.317 


0 


615 


0.693 


1.000 




0.389 


0.332 


0.246 


0 


527 


0.588 „ 


0.626 


1.000 


0.221 


0.288 


0.258 


0.151 


0 


424 


0.50E- 


—0.513 


0.524 


0.395 ' 


0^77 


0.452 


0.182 


0 


430 


0.461 


0.379 


0.377 


0.364 


0.380 


0.#39 


• 0.205 


0 


332 


(J. 370 


0.303 


0.355 


0.362 


0.344 


0.332 


0.191 


0 


376 


0.354 


0.280 


0.247 


0.325 


0.304 


0.324 


0.115 


0. 


396 


0.416 


0.251 


(J. 347 
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APPENDIX D: f Correlation Matrix Among Subscale Scores (continued) 
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0*321 


0.329 


0.362 


0.338 


0.350 


0.135 


0.364 


0.398 


0. 362 


0.302 


27 


0.394 


0.341 


0.417 


0.29* 


0.357 


0.203 


0.318 


0.359 


0.285 


0.262 


28 


0.303 


0.330 


0.250- 


0.277 


0.337* 


0.164 


0.324 


0.352 


0.239 


0.258 


29 


0.330 


0.364 


0.327 


0.355 


0.410 


0.206 


0.286 


0.425 


0.329 


' 0.295 


30 
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0.206 
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0.427 
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1.000 
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0.320 


0.502 


1.000 
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0.211 
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0.4JJ1 


1.000 
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0.274 
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0.509 
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1.000 
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0.464 
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1.000 
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0.430 
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0.482 
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